The new features of AbiPy v0.9.1

M. Giantomassi and the AbiPy group

10th international ABINIT developer workshop
May 31 - June 4, 2021 - Smart Working, Lockdown

---

• These slides have been generated using jupyter, nbconvert and revealjs
• The notebook can be downloaded from this github repo
• To install and configure the software, follow these installation instructions

Use the Space key to navigate through all slides.

What is AbiPy?

Python package for:

• Generating ABINIT input files automatically
• Post-processing output results (netcdf and text files)
• Interfacing ABINIT with external tools (wannier90, lobster, etc)
• Generating and executing ABINIT-specific workflows on laptops as well as clusters

Dependencies:

• Hard deps: pymatgen, netcdf4, matplotlib, plotly, numpy, scipy, pandas …
• Soft deps: ASE, phonopy, ipython, jupyter …

NB: AbiPy can be interfaced with other packages (e.g ASE, phonopy) via converters.

What's new at the level of the installation procedure?

• AbiPy now requires python >= 3.6
• conda packages for AbiPy and ABINIT are now provided by conda-forge
• AbiPy v1.0 will drop support for Abinit8
• We plan to provide recipes for EasyBuild and Spack
• We seek for volunteers to support other package managers (homebrew, apt-get, etc)

How to install AbiPy

Using conda and the conda forge channel (recommended):

    conda install abipy --channel conda-forge

Recipes for AbiPy and ABINIT are now supported by the conda-forge community thus:

• conda packages are automatically generated when new ABINIT/AbiPy releases are pushed to github
• the abiconda channel is deprecated. Please use the conda-forge versions

Since conda is not limited to py packages, one can install ABINIT in the same env with:

    conda install abinit -c conda-forge

Using pip and python wheels:

    pip install abipy --user

From the github repository:

    git clone https://github.com/abinit/abipy.git
    cd abipy 
    python setup.py install # or `develop` for development

---

For further info see http://abinit.github.io/abipy/installation.html

What's new at the level of the documentation?

• New website based on Read-the-Docs theme
• Gallery of plots with matplotlib and plotly examples (XXX scripts in total)
• New examples for Flows (XXX scripts in total)
• Each example can be executed inside a Docker repo on mybinder.org
• New examples in the ABINIT EPH tutorials.

New AbiPy website based on Read-the-Docs theme

%embed https://abinit.github.io/abipy/index.html

#%embed https://abinit.github.io/abipy/gallery/index.html
#%embed https://abinit.github.io/abipy/flow_gallery/index.html

What's new at the level of workflow infrastructure?

• Python converters: DDB $\;\rightleftarrows \;$ phonopy / tdep

• New Flows and improved post-processing tools for:

  • elastic and piezoelectric tensors with DFPT (clamped/relaxed atoms)
  • Gruneisen parameters with DFPT + finite differences
  • quasi-harmonic approximation with DFPT
  • effective masses with DFPT or finite differences
  • isotropic superconducting $T_c$ with Mc Millan equation
  • e-ph self-energy, e-ph matrix elements and scattering potentials
  • phonon-lmited transport properties

in addition to the standard workflows/tools for GS, DFPT, $GW$, IP optics, Bethe-Salpeter, etc.

Note that there are two different workflow infrastructures:

Internal AbiPy implementation (abipy.flowtk modules):

• ✅ Lightweigth, no database is required
• ✅ Designed for rapid prototyping and for testing advanced ABINIT capabilities
• ❌ No explicit support for high-throughput (HT) applications.

AbiFlows package (requires Fireworks and MongoDB database)

• ✅ HT-oriented: can use MongoDB to store workflow status and final results for further analysis
• ✅ High-level API designed for HT applications (e.g. phonon calculations for the materials project
• ❌ New ABINIT features are first implemented/tested with flowtk and then ported to AbiFlows.

NB: There's an ongoing effort to reimplement AbiFlows in terms of the atomate framework. In this talk will mainly discuss the new features available in abipy.flowtk.

DDB converters

(contributed by G. Petretto)

• Based on previous work by H. Xue, E. Bousquet and A. Romero

• Can be used to:

  • connect ABINT DPT part with other packages requiring phonony (e.g. anharmonic calculations hipive)
  • interface AbiPy with phonopy tools e.g. irreps

• Algorithm:

  • Runs anaddb to get the interatomic force constants IFC($\bf{R}$), BECs and $\epsilon^\infty$
  • Save results in nc format (anaddb.nc)
  • Convert IFC($\bf{R}$) and tensors from ABINIT to phonopy conventions
  • Optionally saves standard phonopy files in output_dir_path.

In python, everything boils down to:

ddb = abilab.abiopen("mp-149_DDB")
ph = ddb.anaget_phonopy_ifc(output_dir_path="output_dir")  # Phonopy object

* Example: obtain phonon irreps from DDB using phonopy

```python
ddb = abilab.abiopen("mp-149_DDB")
ph = ddb.anaget_phonopy_ifc(ngqpt=[1,1,1])
ph.set_irreps([0, 0, 0])
ph.get_irreps().show()
```

ElasticWork

This flow computes:

• the rigid-atom elastic tensor
• the rigid-atom piezoelectric tensor (insulators only)
• the internal strain tensor
• the atomic relaxation corrections to the elastic and piezoelectric tensor

For the formalism, see:

Python example

scf_input = make_scf_input() # Build input for GS calculation

elast_work = flowtk.ElasticWork.from_scf_input(scf_input, 
                                               with_relaxed_ion=True, 
                                               with_piezo=True)

Dependency Graph:

---

• The partial DDB files are automaticall merged.
• Final DDB produced in the outdata directory

To generate a GS input, one can use factory functions:

from abipy.abio.factories import gs_input
gs_input(structure="si.cif", pseudos="14si.pspnc", ecut=8)

##############################################
#### SECTION: basic
##############################################
ecut 8
ngkpt 8 8 8
shiftk 0.5 0.5 0.5
nshiftk 1
kptopt 1
nsppol 2
nband 16
occopt 3
tolvrs 1e-08
##############################################
#### SECTION: gstate
##############################################
spinat
0.0 0.0 0.6
0.0 0.0 0.6
chksymbreak 0
nspinor 1
nspden 2
charge 0.0
tsmear 0.0036749322175655 Ha
##############################################
#### STRUCTURE
##############################################
natom 2
ntypat 1
typat 1 1
znucl 14
xred
0.0000000000 0.0000000000 0.0000000000
0.2500000000 0.2500000000 0.2500000000
acell 1.0 1.0 1.0
rprim
6.3285005244 0.0000000000 3.6537614813
2.1095001748 5.9665675141 3.6537614813
0.0000000000 0.0000000000 7.3075229627

or build the GS input explicitly if full control is wanted:

multi = abilab.ebands_input(structure="si.cif", 
                            pseudos="14si.pspnc",
                            ecut=8, 
                            spin_mode="unpolarized", 
                            smearing=None, 
                            dos_kppa=5000)

multi.get_vars_dataframe("kptopt", "iscf", "ngkpt")

	kptopt	iscf	ngkpt
dataset 0	1	None	[8, 8, 8]
dataset 1	-11	-2	None
dataset 2	1	-2	[14, 14, 14]

In order to run the flow, we need two configurations files:

1. scheduler.yml providing:

   • scheduler parameters such as days, hours, minutes, max_njobs_inqueue, max_ncores_used, …

1. manager.yml providing:

   • list of shell commands to be executed before running ABINIT
   • list of modules to load
   • options passed to the queue manager (bluegene, moab, pbspro, sge, shell, slurm, torque)
   • See

See this page for examples or use the abidoc.py script:

• abidoc.py scheduler
• abidoc.py manager
• abidoc.py manager slurm

How to run calculations?

The simplest way to start the scheduler from the shell is through the syntax:

run_elastic.py --scheduler # -s for the short option

For non-trivial Flows, we suggest the scheduler in background and use nohup

nohup run_elastic.py --s > log 2> err &

so that we can disconnect from the shell session without killing the scheduler.

To compute the elastic tensors from the DDB file, use:

elastic_ddb = abilab.abiopen("elastic_DDB")
edata = elastic_ddb.anaget_elastic()

# Pandas DataFrame with relaxed-atoms elastic tensor.
edata.elastic_relaxed

	xx	yy	zz	yz	xz	xy
Voigt index
xx	135.262182	54.450376	38.052927	0.00000	0.000000	0.000000
yy	54.450376	135.262181	38.052927	0.00000	0.000000	0.000000
zz	38.052927	38.052926	148.211029	0.00000	0.000000	0.000000
yz	0.000000	0.000000	0.000000	30.55071	0.000000	0.000000
xz	0.000000	0.000000	0.000000	0.00000	30.550709	0.000000
xy	0.000000	0.000000	0.000000	0.00000	0.000000	40.405903

To compute elastic properties (rigid/relaxed atoms)

edata.get_elastic_properties_dataframe(properties_as_index=True)

	property	0	1
0	trans_v	3.194459e+03	3.838052e+03
1	long_v	5.796035e+03	6.295200e+03
2	snyder_ac	5.767044e+01	8.693459e+01
3	snyder_opt	3.158141e-01	3.621134e-01
4	snyder_total	5.798626e+01	8.729670e+01
5	clarke_thermalcond	7.734051e-01	9.006348e-01
6	cahill_thermalcond	8.539415e-01	9.791319e-01
7	debye_temperature	3.760623e+02	4.477874e+02
8	k_voigt	7.553865e+01	7.553930e+01
9	k_reuss	7.553074e+01	7.553579e+01
10	k_vrh	7.553469e+01	7.553754e+01
11	g_voigt	3.951341e+01	5.767416e+01
12	g_reuss	3.761300e+01	5.366047e+01
13	g_vrh	3.856321e+01	5.566731e+01
14	universal_anisotropy	2.527308e-01	3.740360e-01
15	homogeneous_poisson	2.818554e-01	2.041909e-01
16	y_mod	9.886491e+10	1.340681e+11

For additional info, please consult this notebook tutorial

The mode Gr\"uneisen parameters are the logarithmic derivatives of the phonon frequencies with respect to the volume $V$:

$ \renewcommand{\qq}{{\mathbf{q}}} \newcommand{\qnu}{{\qq\nu}} \newcommand{\wqnu}{{\omega_\qnu}} \newcommand{\PDER}[2]{\dfrac{\partial #1}{\partial #2}} \newcommand{\eqnu}{{\epsilon_\qnu}} $

Gruneisen parameters with finite differences

This flow computes the Gruneisen parameters:

\begin{equation*} \newcommand{\PDER}[2]{\dfrac{\partial #1}{\partial #2}} \gamma_{\mathbf{q}\nu} = -\PDER{\ln \wqnu}{\ln V} = -\dfrac{V}{\wqnu}\,\PDER{\wqnu}{V} \end{equation*}

where $D(\mathbf{q})\,{\epsilon_{\mathbf{q}\nu}} = \omega^2_s(\mathbf{q})\,\epsilon_{\mathbf{q}\nu}$ with $D$ the dynamical matrix.

The derivative wrt $V$ can be rewritten as:

\begin{equation*} \PDER{\omega^2(\mathbf{q}\nu)}{V} = 2\wqnu\,\PDER{\wqnu}{V} \langle \eqnu | \PDER{D(\qq)}{V} | \eqnu \rangle \end{equation*}

where the r.h.s is computed by anaddb using finite differences of DFPT results.

Python example:

scf_input = make_scf_input() # Build input for GS calculation

# NB: k-mesh in gs_inp and ngqpt q-mesh must be commensurate.
from abipy.flowtk.gruneisen import GruneisenWork
voldelta = gs_inp.structure.volume * 0.02
work = GruneisenWork.from_gs_input(gs_inp, voldelta, ngqpt=[2, 2, 2], 
                                   with_becs=False)

Dependency Graph:

---

• $N$ relaxations at fixed volume (violet task). $N$ in [3, 5, 7]
• Each relaxation task starts a DFPT computation of the dynamical matrix $D(\mathbf{q}, V)$
• Finally, invoke anaddb with the $N$ DDB files to produce out_GRUNS.nc

To analyze the results stored in the GRUNS.nc file, use:

gruns = abiopen("out_GRUNS.nc")
gruns.plot_gruns_scatter();

To visualize all the results inside the browser, use

abiopen.py out_GRUNS.nc --expose-web # -ew for the short version

---

This is new feature of abiopen.py made possible by panel. See next slides.

Flow for non-linear optical properties

(contributed by F. Naccarato)

This flow computes:

• SCF + NSCF along k-path to find band edges automatically
• See XXX talk for applications

For the formalism see:

Python example:

scf_input = make_scf_input()   # Get the SCF input (without SOC)

scf_input = make_scf_input()
return flowtk.NonLinearCoeffFlow.from_scf_input("nlo_flow", scf_input)

Dependency Graph:

---

• Foo
• Bar

Effective masses with DFPT

This flow performs:

• SCF followed by NSCF run along k-path to find band edges automatically
• Compute $\epsilon^{\alpha\beta}_{n\bf{k}}$ and the effective mass tensor for these k-points from: $|u_{n\mathbf{k}}\rangle$, and the k-derivatives $|u_{n\mathbf{k}}^\alpha\rangle$ $H^\alpha_{\mathbf{k}}$, $H^{\alpha\beta}_{\mathbf{k}}$

For the formalism see J. Laflamme Janssen, et. al. Phys. Rev. B 93, 205147

Python example:

scf_input = make_scf_input()   # Get the SCF input (without SOC)

from abipy.flowtk.effmass_works import EffMassAutoDFPTWork
flow = flowtk.Flow("effmass_flow")

work = EffMassAutoDFPTWork.from_scf_input(scf_input)
flow.register_work(work)

Dependency Graph

---

• SCF + NSCF along high-symmetry k-path to find band edges automatically
• Perform DFPT computation of $m^*$ tensor for the band edges.
• Similar Flows for effective masses with finite differences are available as well.

e-ph matrix elements along an arbitrary q-path

This flow computes:

This example shows how to compute the e-ph matrix elements in AlAs along a q-path with AbiPy flows. The final results are stored in the GKQ.nc file (one file for q-point) in the outdata of each task.

Python example:

# Build input for GS calculation on a 2x2x2 k-mesh
scf_input = make_scf_input(ngkpt=(2, 2, 2))

# q-mesh for phonons
ngqpt = (2, 2, 2)

# Create flow to compute all the independent atomic perturbations
# Use ndivsm = 0 to pass an explicit list of q-points.
# If ndivsm > 0, qpath_list is interpreted as a list of boundaries for the q-path
qpath_list = [[0.0, 0.0, 0.0], [0.01, 0, 0], [0.1, 0, 0],
              [0.24, 0, 0], [0.3, 0, 0], [0.45, 0, 0], [0.5, 0.0, 0.0]]

from abipy.flowtk.eph_flows import GkqPathFlow
flow = GkqPathFlow.from_scf_input("flow_dir", scf_input,
                                  ngqpt, qpath_list, ndivsm=0, with_becs=True,
                                  ddk_tolerance={"tolwfr": 1e-8})

Formalism:

https://abinit.github.io/abipy/flow_gallery/run_gkq.html#sphx-glr-flow-gallery-run-gkq-py

Dependency Graph:

---

• SCF + NSCF along k-path to find band edges automatically
• Used in ...

Other applications of AbiPy flows are discussed in the following talks:

• Electron-phonon beyond Fröhlich: dynamical quadrupoles in polar and covalent solids by G. Brunin
• Phonon-limited conductivity in 2D and 3D metals by O. Nadeau
• Absorption spectrum calculations using cumulant expansion in electron-phonon interactions by J. C. Abreu
• Automating ΔSCF computations of point defects using AbiPy workflows by J. Bouquiaux

What's new at the level of the post-processing tools?

• New plotting tools based on plotly
• GUIs and dashboards based on panel and bokeh

Why plotly?

• ✅ Publication quality figures.
• ✅ Flexible python API able to produce rather advanced plots
• ❌ Plots are difficult to customize without changing the python code
• ❌ Plots lacks interactivity and integration with HTML/JS

• ✅ Interactive plots + chart editor GUI to customize the figure
• ✅ Play well with HTML (plotly us written in js with python bindings)
• ❌ Open source project but not all the features are available in the free plan
• ❌ Requires browser (this may represent an issue on some HPC centers)

Important note

• AbiPy will continue to support and develop matplotlib-based tools
• We plan to implement plotly-based tools for the most important physical properties but we cannot support all the possible plots already implemented with matplotlib
• Fortunately, there are packages such as holoviz or plotly-express that provided the data is stored in a pandas dataframe.

Several other plotting libraries are available in the pydata ecosystem

AbiPy plots with matplotlib

gsr = abiopen("si_nscf_GSR.nc")
gsr.ebands.plot(with_gaps=True);

AbiPy plots with plotly

(contributed by Y. He and MG)

gsr.ebands.plotly(with_gaps=True);  # object.plot becomes object.plotly

To update the plotly figure to the chart studio server, use:

gsr.ebands.plotly(with_gaps=True, chart_studio=True);

---

Users can finally customize the AbiPy plot without changing the py code 🎉

Interactive 3d plots with plotly:

gsr.ebands.kpoints.plotly(title="k-path in 3d with plotly");

The matplotlib 3d plot embedded in HTML :

gsr.ebands.kpoints.plot(title="Static matplotlib figure");

Integrating Abipy with web-based technologies

• Integration between panel and AbiPy
• AbiPy GUIs inside jupyter notebooks
• Dashboards and web apps
• Integration with the AbiPy command line interface

What is Panel?

• Panel provides tools for composing widgets, plots, tables into apps and dashboards
• It relies on the client-server model where:
  • the client is the web browser running HTML/CSS/JS code.
  • the server communicates with the client, executes py code and sends the results back to the client
• In a nutshell, panel keeps the browser and python in synch e.g. the user clicks a button in the GUI and the signal to python.

Why panel?

• Panel works with visualizations from Bokeh, Plotly, Matplotlib, HoloViews, and many other Python plotting libraries,
• Panel works equally well in Jupyter Notebooks

Thanks to this infrastructure, developers can:

• use python to solve the scientific problem using the pydata ecosystem and other ab-inition py packages
• use modern web-technologies to present the results to the client without having to deal with HTML/CSS/JS

Pros and cons of the client-server model

Advantages:

• Client does not need to install the scientific software stack (when running on different machines)
• Can take advantage of modern HTML technologies to present the results in the browser
• Can implement web apps that allows the user to upload the data and analyze the results withouth having to install python software.

Disavantages

• Round trip delay if client != host and slow connection
• Upoloadng a 1Gb file to the remote server just because you don't want to install software on the localhost is a very bad idea.
• OK for relatively small files (< 1Gb) but this approach is not designed to handle big data.
• Not all the HPC centers provide specialized nodes to post-process the results inside a web browser/notebook.

#gsr = abiopen("si_nscf_GSR.nc")
#abilab.abipanel();
#gsr.get_panel()

How to use AbiPy panels inside jupyter notebooks

To build a panel GUI inside the notebook, call the get_panel method:

ddb = abilab.abiopen("ZnSe_hex_qpt_DDB")

abilab.abipanel(); # Important
ddb.get_panel()

Other AbiPy files/objects are supported as well:

ddb.structure.get_panel()

Don't be surprised if you start to click buttons and nothing happens in the GUI! One needs a **running python backend** to execute the callbacks triggerered by the widgets.

Creating dashboards from the command line

• Calling get_panel inside a notebook is great as you have GUIs and at the same time you can execute python code.
• There are however cases in which we only need a dahhboard with widgets to interact with the data.

To launch the dashboard associated to the DDB file from the shell, use the --panel option:

abiopen.py out_DDB --panel  # or -pn if you preder the short option.

---

To produce a predefined set of matplotlib figures, use:

abiopen.py mgb2_kpath_FATBANDS.nc --expose --seaborn

abiopen.py mgb2_kpath_FATBANDS.nc --expose-web # -ewb

abiopen.py mgb2_kpath_FATBANDS.nc --plotly # -ply

Creating dashboards without widgets with abiview

to get a quick look at the results

abiview.py ddb ZnSe_hex_qpt_DDB --panel  # or -pn if you prefer the short version.

Need to call anaddb to compute and visualize ph-bands and DOS from DDB?

abiview.py ddb ZnSe_hex_qpt_DDB --plotly # -ply

Documentation for these new features available at:

%embed https://abinit.github.io/abipy/graphical_interface.html

#plotter = abilab.ElectronBandsPlotter()
#plotter.add_ebands(label="BZ sampling", bands="si_scf_GSR.nc")
#plotter.add_ebands(label="k-path", bands="si_nscf_GSR.nc")
#plotter.gridplot(with_gaps=True);

Future developments

Post-processing tools

• Implement panel GUIs for all the netcdf files already supported by AbiPy
• Develop web applications so that users can upload e.g. DDB files and analyze the results

Continuous Integration

• Use AbiPy programmatic interface to implement:

  • Validation of parallel algorithms for np in range(1, N)
  • Stress testing
  • Benchmarks